Learning policies for Markov decision processes from data

We consider the problem of learning a policy for a Markov decision process consistent with data captured on the state-actions pairs followed by the policy. We assume that the policy belongs to a class of parameterized policies which are defined using features associated with the state-action pairs. The features are known a priori, however, only an unknown subset of them could be relevant. The policy parameters that correspond to an observed target policy are recovered using `1-regularized logistic regression that best fits the observed state-action samples. We establish bounds on the difference between the average reward of the estimated and the original policy (regret) in terms of the generalization error and the ergodic coefficient of the underlying Markov chain. To that end, we combine sample complexity theory and sensitivity analysis of the stationary distribution of Markov chains. Our analysis suggests that to achieve regret within order O( √ ), it suffices to use training sample size on the order of Ω(logn · poly(1/ )), where n is the number of the features. We demonstrate the effectiveness of our method on a synthetic robot navigation example

Learning policies for Markov decision processes from data

We consider the problem of learning a policy for a Markov decision process consistent with data captured on the state-actions pairs followed by the policy. We assume that the policy belongs to a class of parameterized policies which are defined using features associated with the state-action pairs. The features are known a priori, however, only an unknown subset of them could be relevant. The policy parameters that correspond to an observed target policy are recovered using `1-regularized logistic regression that best fits the observed state-action samples. We establish bounds on the difference between the average reward of the estimated and the original policy (regret) in terms of the generalization error and the ergodic coefficient of the underlying Markov chain. To that end, we combine sample complexity theory and sensitivity analysis of the stationary distribution of Markov chains. Our analysis suggests that to achieve regret within order O( √ ), it suffices to use training sample size on the order of Ω(logn · poly(1/ )), where n is the number of the features. We demonstrate the effectiveness of our method on a synthetic robot navigation example

Encounter complexes and dimensionality reduction in protein-protein association

An outstanding challenge has been to understand the mechanism whereby proteins associate. We report here the results of exhaustively sampling the conformational space in protein–protein association using a physics-based energy function. The agreement between experimental intermolecular paramagnetic relaxation enhancement (PRE) data and the PRE profiles calculated from the docked structures shows that the method captures both specific and non-specific encounter complexes. To explore the energy landscape in the vicinity of the native structure, the nonlinear manifold describing the relative orientation of two solid bodies is projected onto a Euclidean space in which the shape of low energy regions is studied by principal component analysis. Results show that the energy surface is canyon-like, with a smooth funnel within a two dimensional subspace capturing over 75% of the total motion. Thus, proteins tend to associate along preferred pathways, similar to sliding of a protein along DNA in the process of protein-DNA recognition

Massive hematuria due to a congenital renal arteriovenous malformation mimicking a renal pelvis tumor: a case report

<p>Abstract</p> <p>Introduction</p> <p>Congenital renal arteriovenous malformations (AVMs) are very rare benign lesions. They are more common in women and rarely manifest in elderly people. In some cases they present with massive hematuria. Contemporary treatment consists of transcatheter selective arterial embolization which leads to resolution of the hematuria whilst preserving renal parenchyma.</p> <p>Case presentation</p> <p>A 72-year-old man, who was heavy smoker, presented with massive hematuria and flank pain. CT scan revealed a filling defect caused by a soft tissue mass in the renal pelvis, which initially led to the suspicion of a transitional cell carcinoma (TCC) of the upper tract, in view of the patient's age and smoking habits. However a subsequent retrograde study could not depict any filling defect in the renal pelvis. Selective right renal arteriography confirmed the presence of a renal AVM by demonstrating abnormal arterial communication with a vein with early visualization of the venous system. At the same time successful selective transcatheter embolization of the lesion was performed.</p> <p>Conclusion</p> <p>This case highlights the importance of careful diagnostic work-up in the evaluation of upper tract hematuria. In the case presented, a congenital renal AVM proved to be the cause of massive upper tract hematuria and flank pain in spite of the initial evidence indicating the likely diagnosis of a renal pelvis tumor.</p

Optimal static pricing for a tree network

We study the static pricing problem for a network service provider in a loss system with a tree structure. In the network, multiple classes share a common inbound link and then have dedicated outbound links. The motivation is from a company that sells phone cards and needs to price calls to different destinations. We characterize the optimal static prices in order to maximize the steady-state revenue. We report new structural findings as well as alternative proofs for some known results. We compare the optimal static prices versus prices that are asymptotically optimal, and through a set of illustrative numerical examples we show that in certain cases the loss in revenue can be significant. Finally, we show that static prices obtained using the reduced load approximation of the blocking probabilities can be easily obtained and have near-optimal performance, which makes them more attractive for applications.Massachusetts Institute of Technology. Center for Digital BusinessUnited States. Office of Naval Research (Contract N00014-95-1-0232)United States. Office of Naval Research (Contract N00014-01-1-0146)National Science Foundation (U.S.) (Contract DMI-9732795)National Science Foundation (U.S.) (Contract DMI-0085683)National Science Foundation (U.S.) (Contract DMI-0245352

Future cosmological evolution in f(R)f(R) gravity using two equations of state parameters

We investigate the issues of future oscillations around the phantom divide for $f(R)$ gravity. For this purpose, we introduce two types of energy density and pressure arisen from the $f(R)$-higher order curvature terms. One has the conventional energy density and pressure even in the beginning of the Jordan frame, whose continuity equation provides the native equation of state $w_{\rm DE}$. On the other hand, the other has the different forms of energy density and pressure which do not obviously satisfy the continuity equation. This needs to introduce the effective equation of state $w_{\rm eff}$ to describe the $f(R)$-fluid, in addition to the native equation of state $\tilde{w}_{\rm DE}$. We confirm that future oscillations around the phantom divide occur in $f(R)$ gravities by introducing two types of equations of state. Finally, we point out that the singularity appears ar $x=x_c$ because the stability condition of $f(R)$ gravity violates.Comment: 23 pages, 10 figures, correcting typing mistake in titl

Constraint propagation equations of the 3+1 decomposition of f(R) gravity

Theories of gravity other than general relativity (GR) can explain the observed cosmic acceleration without a cosmological constant. One such class of theories of gravity is f(R). Metric f(R) theories have been proven to be equivalent to Brans-Dicke (BD) scalar-tensor gravity without a kinetic term. Using this equivalence and a 3+1 decomposition of the theory it has been shown that metric f(R) gravity admits a well-posed initial value problem. However, it has not been proven that the 3+1 evolution equations of metric f(R) gravity preserve the (hamiltonian and momentum) constraints. In this paper we show that this is indeed the case. In addition, we show that the mathematical form of the constraint propagation equations in BD-equilavent f(R) gravity and in f(R) gravity in both the Jordan and Einstein frames, is exactly the same as in the standard ADM 3+1 decomposition of GR. Finally, we point out that current numerical relativity codes can incorporate the 3+1 evolution equations of metric f(R) gravity by modifying the stress-energy tensor and adding an additional scalar field evolution equation. We hope that this work will serve as a starting point for relativists to develop fully dynamical codes for valid f(R) models.Comment: 25 pages, matches published version in CQG, references update

Lunar Hydrospheric Explorer (HYDROX)

The Lunar Hydrospheric Explorer (HYDROX) is a 6U CubeSat designed to further confirm the existence of lunar exospheric water, and to determine source processes and surface sites, through ion mass spectrometer measurements of water group (O+, OH+, H2O+) and related ions at energy charge up to 2 keV/e. and mass/charge 1-40amu/e. HYDROX would follow up on the now-concluded exospheric compositional measurements by the Neutral Mass Spectrometer on the NASA LADEE mission and on other remote sensing surface and exospheric measurements (LADEE,LRO, etc.)

The Energetic Particle Detector (EPD) Investigation and the Energetic Ion Spectrometer (EIS) for the Magnetospheric Multiscale (MMS) Mission

Abstract The Energetic Particle Detector (EPD) Investigation is one of 5 fields-and-particles investigations on the Magnetospheric Multiscale (MMS) mission. MMS comprises 4 spacecraft flying in close formation in highly elliptical, near-Earth-equatorial orbits targeting understanding of the fundamental physics of the important physical process called magnetic reconnection using Earth’s magnetosphere as a plasma laboratory. EPD comprises two sensor types, the Energetic Ion Spectrometer (EIS) with one instrument on each of the 4 spacecraft, and the Fly’s Eye Energetic Particle Spectrometer (FEEPS) with 2 instruments on each of the 4 spacecraft. EIS measures energetic ion energy, angle and elemental compositional distributions from a required low energy limit of 20 keV for protons and 45 keV for oxygen ions, up to \u3e0.5 MeV (with capabilities to measure up to \u3e1 MeV). FEEPS measures instantaneous all sky images of energetic electrons from 25 keV to \u3e0.5 MeV, and also measures total ion energy distributions from 45 keV to \u3e0.5 MeV to be used in conjunction with EIS to measure all sky ion distributions. In this report we describe the EPD investigation and the details of the EIS sensor. Specifically we describe EPD-level science objectives, the science and measurement requirements, and the challenges that the EPD team had in meeting these requirements. Here we also describe the design and operation of the EIS instruments, their calibrated performances, and the EIS in-flight and ground operations. Blake et al. (The Flys Eye Energetic Particle Spectrometer (FEEPS) contribution to the Energetic Particle Detector (EPD) investigation of the Magnetospheric Magnetoscale (MMS) Mission, this issue) describe the design and operation of the FEEPS instruments, their calibrated performances, and the FEEPS in-flight and ground operations. The MMS spacecraft will launch in early 2015, and over its 2-year mission will provide comprehensive measurements of magnetic reconnection at Earth’s magnetopause during the 18 months that comprise orbital phase 1, and magnetic reconnection within Earth’s magnetotail during the about 6 months that comprise orbital phase 2

Error-analysis and comparison to analytical models of numerical waveforms produced by the NRAR Collaboration

The Numerical-Relativity-Analytical-Relativity (NRAR) collaboration is a joint effort between members of the numerical relativity, analytical relativity and gravitational-wave data analysis communities. The goal of the NRAR collaboration is to produce numerical-relativity simulations of compact binaries and use them to develop accurate analytical templates for the LIGO/Virgo Collaboration to use in detecting gravitational-wave signals and extracting astrophysical information from them. We describe the results of the first stage of the NRAR project, which focused on producing an initial set of numerical waveforms from binary black holes with moderate mass ratios and spins, as well as one non-spinning binary configuration which has a mass ratio of 10. All of the numerical waveforms are analysed in a uniform and consistent manner, with numerical errors evaluated using an analysis code created by members of the NRAR collaboration. We compare previously-calibrated, non-precessing analytical waveforms, notably the effective-one-body (EOB) and phenomenological template families, to the newly-produced numerical waveforms. We find that when the binary's total mass is ~100-200 solar masses, current EOB and phenomenological models of spinning, non-precessing binary waveforms have overlaps above 99% (for advanced LIGO) with all of the non-precessing-binary numerical waveforms with mass ratios <= 4, when maximizing over binary parameters. This implies that the loss of event rate due to modelling error is below 3%. Moreover, the non-spinning EOB waveforms previously calibrated to five non-spinning waveforms with mass ratio smaller than 6 have overlaps above 99.7% with the numerical waveform with a mass ratio of 10, without even maximizing on the binary parameters.Comment: 51 pages, 10 figures; published versio